Cognitive computer assisted attribute acquisition through iterative disclosure

ABSTRACT

A method, system, and computer program product are provided for displaying query items (e.g., patient attributes) and answers (e.g., treatment recommendations) by performing a ranking analysis of query items by running a cognitive analysis comparison of each marginal answer confidence improvement metric for each unspecified query item in the first plurality of query items to rank the query items in sorted order from largest to smallest marginal answer confidence improvement metric.

BACKGROUND OF THE INVENTION

In the field of artificially intelligent computer systems capable ofanswering questions posed in natural language; cognitive questionanswering (QA) systems (such as the IBM Watson™ artificially intelligentcomputer system and/or other natural language question answeringsystems) process questions posed in natural language to determineanswers and associated confidence scores based on knowledge acquired bythe QA system. Such cognitive QA systems provide powerful tools that canbe used in a variety of different applications or fields, such asfinancial, medical, scientific research, engineering, software, and thelike. While there remain challenges with processing the ever increasingamount of unstructured data (such as, for example, the research data,medical records, clinical trials, etc. in the medical field), there arealso significant challenges with evaluating the processing results, suchas selecting an answer or conclusion from a large, but finite list ofpossibilities gathered through random acquisition of deterministicfactors from a large, finite body of unknown attributes. For example,the decision support process used to make medical treatmentrecommendations often relies on patient attributes as query parametersthat do not effectively reduce the uncertainty of the treatmentrecommendations or answers. While cognitive QA systems can providecomputational power to assimilate and analyze the meaning and context ofstructured and unstructured data (such as clinical notes, reports, andkey patient information) to generate a wealth of candidate treatmentoption recommendations, the clinical decision making process canactually be impaired when the most valuable patient attributes are notused to select treatment recommendations. Existing solutions forcomputer assisted decision-making have been limited to operating withstructured data (e.g., Bayesian Network decision support systems) orhave been narrowly applied (e.g., using correlation engines to mapsymptoms to diseases), but such solutions do not prioritize theinformation acquisition used to optimize the decision-making process insupport of decision outcomes, such as treatment recommendations. As aresult, the existing solutions for efficiently and accurately processingand evaluating queries against large and complex amounts of unstructureddata to improve the quality of generated answers are extremely difficultat a practical level.

SUMMARY

Broadly speaking, selected embodiments of the present disclosure providea system, method, and apparatus for optimizing the acquisition of queryattributes by implementing iterative disclosure techniques with thecognitive power of the information handling system to evaluate userqueries against large unstructured data sets using natural languageprocessing to return responses and associated confidence values whichare used to iteratively guide the user's query submissions to acquirethe most valuable query attributes in ranked order by the number ofpossible conclusions that can be deprioritized b the attribute thatwould significantly improve the selection process. As an initial step inselected embodiments of the present disclosure, the information handlingsystem identifies, retrieves, processes, and/or combines queryattributes from a user (e.g., patient attributes). Evaluating the queryattributes against a large unstructured data set, the informationhandling system returns responses and confidence values, such as byapplying one or more learning methods to identify and rank additionalquery attributes and potential responses or answers (e.g., treatmentplans or recommendations) from unstructured data based on associatedconfidence values. To this end, vector representations of the queryattributes (e.g., patient attribute vectors) may be mapped to vectorrepresentations of the response confidence values (e.g., treatmentconfidence vectors) and computationally processed to derive confidencevolume metrics to represent the confidence uncertainty for a given setof responses and query attributes. Based on the confidence volumemetrics or a normalized representation thereof, the information handlingsystem can prioritize additional query attribute information to beprovided by the user to most effectively reduce the uncertainty in theconfidence values for a set of responses or answers. For example, theconfidence volume metric may be computed as the product of optionconfidence ranges for each query attribute or piece of information, andthen each query attribute or information can be ranked based on amarginal reduction in the confidence volume metric.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations, and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, inventive features, and advantages of the present invention, asdefined solely by the claims, will become apparent in the non-limitingdetailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings, wherein:

FIG. 1 depicts a system diagram of an information handling systemconnected in a network environment that uses an iterative attributeacquisition engine to optimize and guide the iterative acquisition ofquery attributes for use in generating treatment recommendations;

FIG. 2 is a block diagram of a processor and components of aninformation handling system such as shown in FIG. 1;

FIG. 3 is a simplified block diagram flow chart showing the logic forusing a machine learning process to iteratively acquire query attributesand generate a list of ranked answers;

FIG. 4 illustrates a first mapping of attribute vectors to a confidencevector space where a specific attribute vector is known;

FIG. 5 illustrates a second mapping of attribute vectors to a confidencevector space where none of the attribute vector values are known;

FIG. 6 illustrates a third mapping of attribute vectors to a confidencevector space where only some of the attribute vector values are knownand others are unknown; and

FIGS. 7-11 illustrate a sequence of display screens to illustrate theiterative acquisition of patient attributes and resulting treatmentrecommendations based on computed confidence volume, customer caremetric (CCM), and marginal CCM increase values which are used tooptimally rank and display the patient attributes and treatment optionspresented to the user.

DETAILED DESCRIPTION

The present invention may be a system, a method, and/or a computerprogram product. In addition, selected aspects of the present inventionmay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and/or hardware aspects thatmay all generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present invention may take theform of computer program product embodied in a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention,Thus embodied, the disclosed system, method, and/or a computer programproduct are operative to improve the functionality and operation of acognitive question answering (QA) system by optimizing theidentification of query attributes which will improve the confidencescores for answers generated by the cognitive QA system, therebyproviding a more efficient and accurate decision-making interface.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a dynamic or static random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a magnetic storage device, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing, Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media light pulses passingthrough a fiber-optic cable), or electrical signals transmitted througha wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server or cluster of servers. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

FIG. 1 depicts a schematic diagram of one illustrative embodiment of aquestion/answer (QA) system 1 connected across a computer network 9 to aplurality of computing devices (e.g., 120, 130, 140), where the QAsystem 100 uses an iterative attribute acquisition engine 11 to optimizeand guide the iterative acquisition of query attributes that are used togenerate treatment recommendations. The QA system 1 may include one ormore QA system pipelines 1A, 1B, each of which includes a knowledgemanager computing device 2 (comprising one or more processors and one ormore memories, and potentially any other computing device elementsgenerally known in the art including buses, storage devices,communication interfaces, and the like) for processing questionsreceived over the network 9 from one or more users at computing devices(e.g., 120, 130, 140). Over the network 9, the computing devicescommunicate with each other and with other devices or components via oneor more wired and/or wireless data communication links, where eachcommunication link may comprise one or more of wires, routers, switches,transmitters, receivers, or the like. In this networked arrangement, theQA system 1 and network 9 may enable question/answer (QA) generationfunctionality for one or more content users. Other embodiments of QAsystem 1 may be used with components, systems, sub-systems, and/ordevices other than those that are depicted herein.

In the QA system 1, the knowledge manager 2 may be configured to receiveinputs from various sources. For example, knowledge manager 2 mayreceive input from the network 9, one or more knowledge databases orcorpora 4 of electronic documents 5, semantic data 6, or other data,content users, and other possible sources of input, such as diagnosisand/or medical data 7 or patient profile/preference data 8. In selectedembodiments, the knowledge base 4 may include structured,semi-structured, and/or unstructured content in a plurality of documentsthat are contained in one or more large knowledge databases or corpora.The various computing devices (e.g., 120, 130, 140) on the network 9 mayinclude access points for content creators and content users. Some ofthe computing devices may include devices for a database storing thecorpus of data as the body of information used by the knowledge manager2 to generate answers to cases. The network 9 may include local networkconnections and remote connections in various embodiments, such thatknowledge manager 2 may operate in environments of any size, includinglocal and global, e.g., the Internet. Additionally, knowledge manager 2serves as a front-end system that can make available a variety ofknowledge extracted from or represented in documents, network-accessiblesources and/or structured data sources. In this manner, some processespopulate the knowledge manager with the knowledge manager also includinginput interfaces to receive knowledge requests and respond accordingly.

In one embodiment, the content creator creates content in an electronicdocument 5 for use as part of a corpora 4 of data with knowledge manager2. The corpora 4 may include any structured and unstructured documents,including but not limited to any file, text, article, or source of data(e.g., scholarly articles, dictionary definitions, encyclopediareferences, and the like) for use in knowledge manager 2. Content usersmay access knowledge manager 2 via a network connection or an Internetconnection to the network 9, and may input questions to knowledgemanager 2 that may be answered by the content in the corpus of data. Aswill be appreciated, when a process evaluates a given section of adocument for semantic content, the process can use a variety ofconventions to query it from the knowledge manager. One convention is tosend a well-formed question 10. Semantic content is content based on therelation between signifiers, such as words, phrases, signs, and symbols,and what they stand for, their denotation, or connotation. In otherwords, semantic content is content that interprets an expression, suchas by using Natural Language (NL) Processing. In one embodiment, theprocess sends well-formed questions 10 (e.g., natural languagequestions, etc.) to the knowledge manager 2. Knowledge manager 2 mayinterpret the question and provide a response to the content usercontaining one or more answers 20 to the question 10. In someembodiments, knowledge manager 2 may provide a response to users in aranked list of answers 20 which include associated confidence values andsupporting evidence.

In some illustrative embodiments, QA system 1 may be the IBM Watson™ QAsystem available from International Business Machines Corporation ofArmonk, N.Y., which is augmented with the mechanisms of the illustrativeembodiments described hereafter. The IBM Watson™ knowledge managersystem may receive an input question 10 which it then parses to extractthe major features of the question, that in turn are then used toformulate queries that are applied to the corpus of data stored in theknowledge base 4. Based on the application of the queries to the corpusof data, a set of hypotheses, or candidate answers to the inputquestion, are generated by looking across the corpus of data forportions of the corpus of data that have some potential for containing avaluable response to the input question.

In particular, a received question 10 may be processed by the IBMWatson™ A system 1 which performs deep analysis on the language of theinput question 10 and the language used in each of the portions of thecorpus of data found during the application of the queries using avariety of reasoning algorithms. There may be hundreds or even thousandsof reasoning algorithms applied, each of which performs differentanalysis, e.g., comparisons, and generates a score. For example, somereasoning algorithms may look at the matching of terms and synonymswithin the language of the input question and the found portions of thecorpus of data. Other reasoning algorithms may look at temporal orspatial features in the language, while others may evaluate the sourceof the portion of the corpus of data and evaluate its veracity.

The scores obtained from the various reasoning algorithms indicate theextent to which the potential response is inferred by the input questionbased on the specific area of focus of that reasoning algorithm. Eachresulting score is then weighted against a statistical model. Thestatistical model captures how well the reasoning algorithm performed atestablishing the inference between two similar passages for a particulardomain during the training period of the IBM Watson™ QA system. Thestatistical model may then be used to summarize a level of confidencethat the IBM Watson™ QA system has regarding the evidence that thepotential response, i.e., candidate answer, is inferred by the question.This process may be repeated for each of the candidate answers until theIBM Watson™ QA system identifies candidate answers that surface as beingsignificantly stronger than others and thus, generates a final answer,or ranked set of answers, for the input question. The QA system 1 thengenerates an output response or answer 20 with the final answer andassociated confidence and supporting evidence. More information aboutthe IBM Watson™ QA system may be obtained, for example, from the IBMCorporation website, IBM Redbooks, and the like. For example,information about the IBM Watson™ QA system can be found in Yuan et al.,“Watson and Healthcare,” IBM developerWorks, 2011 and “The Era ofCognitive Systems: An Inside Look at IBM Watson and How it Works” by RobHigh, IBM Redbooks, 2012. Another example QA system 1 is the IBM Watsonfor Oncology™ cognitive computing system which is designed to supportoncology physicians as they consider treatment options with theirpatients by analyzing a patient's medical information against a vastarray of data and expertise to interpret the cancer patients' clinicalinformation and identify individualized, evidence-based treatmentoptions.

To process and answer questions, the QA system 1 may include aninformation handling system 3 which uses an iterative attributeacquisition engine 11 to iteratively guide query submissions to generatetreatment recommendations by assembling patient attributes which aremapped to treatment confidence metrics, submitting the patientattributes as query submissions to the information handling system,evaluating the patient attributes and confidence metrics at theinformation handling system using machine learning techniques toidentify potential treatment plans or recommendations for a patient,selecting additional patient attributes for input as additional querysubmissions which will most effectively reduce the uncertainty in thetreatment confidence metrics, and then continuing to submit the patientattributes as query submissions to the information handling system in aniterative disclosure sequence until one or more treatment plans orrecommendations are selected for display, comparison, and evaluation bythe patient. Though shown as being embodied in or integrated with the QAsystem 1, the information handling system 3 and/or iterative attributeacquisition engine 11 may be implemented in whole or in part in aseparate computing system (e.g., 150) that is connected across a network9 to the QA system 1. Wherever embodied, the cognitive power of theiterative attribute acquisition engine 11 processes input patientattribute data from the physician diagnosis and patient medical data togenerate candidate treatment recommendations, evaluates the inputpatient attribute data on the basis of confidence metrics to generatetreatment outcomes specified by the patient profile, preference, and/orpatient attribute data, and then generates an optimized visualization ofthe treatment recommendations. As will be appreciated, the iterativeattribute acquisition techniques disclosed herein may be used in otherapplications besides making medical treatment recommendations based onpatient attribute data

In selected example embodiments, the iterative attribute acquisitionengine 11 may include an assembly system, module or interface 12 whichuses confidence metric values to assemble query attributes for use initeratively guiding query submissions to the QA system 1. In selectedembodiments, the query attributes may be patient attributes relating toa patient's medical information or data, such as the diagnostic datafrom the physician, clinical expertise, external research, or othermedical data, Examples of such patient medical information or datainclude, but are not limited to basic patient demographic information(e.g., age and gender), the patient's electronic medical record (e.g.,weight, height, blood pressure, vitals, problems/diagnoses, medications,allergies, patient charts, documents, vaccinations, lab results,confidential notes, images, etc.), medical conditions (e.g., dyspnea,hemoglobin, bronchoscopy, blood creatinine, parathyroid hormones, footnumbness, polysomnography, urinalysis, etc.), and/or the patient'sdiagnosis result (e.g., current tumor size or prognosis, etc.). As analternative to providing an assembly system, module or interface 12, thepatient attributes relating to a patient's medical information or datamay have already been stored by the QA system 1 during the patient'sprevious visit. In yet other embodiments, the query attributes may berelated to other fields of investigation, such as finances, medicine,scientific research, engineering, software, and the like. Howeverassembled or received, the assembly system, module or interface 12displays the patient attributes for use in guiding the iterativedisclosure of query submissions, such as by displaying a listing ofpatient attributes ranked by confidence metric values, where each rankedpatient attribute may be individually selected by the user (e.g.,medical technician) for submission of additional data or informationrelating to the selected patient attribute.

Upon receiving or otherwise obtaining the input patient attribute dataor information, the iterative attribute acquisition engine 11 may beconfigured to evaluate the patient attributes and generate treatmentrecommendations based on associated treatment confidence metric values.To this end, the iterative attribute acquisition engine 11 may include atreatment recommendation system, module or interface 13 that isconfigured to apply natural language processing (NLP) techniques, suchas machine learning and/or deep analytic analysis, to evaluate patientattributes that are input by the physician and/or extracted from thepatient's medical data and diagnostic data along with associatedtreatment confidence metric values, and to generate therefrom treatmentrecommendations for this medical case, where each recommended treatmentincludes one or more specified treatment outcomes, such as potentialside-effects, cost, and patient's survival rate. Typically, the outputgenerated by the treatment recommendation system 13 may includemultiple, different treatment options, each having an associatedtreatment confidence metric value and listing of supporting evidence toassist the physician in making informed treatment decisions. However,the large number of treatment options can overwhelm the physician orpatient's decision-making process which must consider the attributes orside effects of the many disparate treatments along with the patient'spersonal medical record information, particularly in the stressfulcontext of making health decisions when the patient is stressed,injured, or sick. In addition, the quality or accuracy of the treatmentrecommendations (as represented by the treatment confidence metricvalues) may be limited by the completeness or accuracy of the availablepatient attribute data. For example, an incomplete assembly of patientattribute data will result in less accurate treatment recommendationsthan would be generated from a complete picture of the patent'sattributes. To improve the assembly of patent attribute data,conventional decision support systems require that the physician orexpert must use their own knowledge and experience to determine whichinformation to obtain in what priority to best improve the confidencescores for all treatment recommendations.

To improve the accuracy and efficiency of generating treatmentrecommendations, the treatment recommendation system, module orinterface 13 may be configured to apply NLP techniques to the patientattributes and associated treatment confidence metric values in order toprioritize and identify the patient attribute information that should beentered to most effectively reduce the uncertainty in the confidencevalues for the treatment recommendations. In selected embodiments, amachine learning processing prioritizes patient attributes orinformation by computing or deriving a treatment confidence volumemetric (which can refer to an area, volume or hyper-value, depending onthe number of attributes) and/or a normalized representation of thetreatment confidence volume metric (also referred to as a Customer CareMeasure (CCM)), both of which provide different measures of theuncertainty in the confidence values for the treatment options provided.

As disclosed herein, the treatment recommendation system, module orinterface 13 may compute the treatment confidence volume metric valuesas the product of option confidence ranges for each patient attribute orpiece of information. The larger the treatment confidence volume metric,the greater the confidence uncertainty about the treatmentrecommendation. And conversely, the smaller the treatment confidencevolume metric, the greater the confidence certainty about the treatmentrecommendation. Once computed, the treatment confidence volume metriccan be used to rank the patient attributes or information based on amarginal reduction in the confidence volume by prioritizing theattributes which will most effectively reduce the confidence uncertaintyabout the treatment recommendation.

As further disclosed herein, the treatment recommendation system, moduleor interface 13 may compute the normalized treatment confidence volumemetric (CCM) value as a number between 0 and 1 with the equationCCM=1−(Vknown/Vempty), where Vknown is the confidence volume for a setof known attributes (represented by the vector a′) and where Vempty isthe confidence volume when no patient attributes are known. Oncecomputed, the CCM value can be used to rank additional patient attributeinput and also to provide a reference point indication of the number ofknown patient attributes.

As further disclosed herein, the treatment recommendation system, moduleor interface 13 may compute a marginal improvement in the customer caremeasure to determine how much uncertainty can be reduced by providingadditional patient attribute data for a selected patient attribute. Inselected embodiments, the marginal increase in the CCM value (MCCM) foran attribute A may be calculated from confidence volume values asMCCM=(V_(A)−Vknown)/Vempty, where V_(A) is the confidence volume for aset of known attributes and one additional attribute A, Vknown is theconfidence volume for a set of known attributes (represented by thevector a′), and where Vempty is the confidence volume when no patientattributes are known.

As indicated by the feedback path 14, the additional patient attributeinformation and treatment confidence volume metric values are fed backto the assembly system, module or interface 12 for use in assembling anddisplaying patient attributes to be used in a guided iterativedisclosure approach for submitting additional queries. With thisfeedback information, the assembly system, module or interface 12 may beconfigured to determine the display priority and visualizationattributes of patient attributes and treatment recommendations based onthe treatment confidence volume metric values computed at the treatmentrecommendation system, module or interface 13. For example, the patientattribute data collected as a first set of patient attribute data duringan initial pass through the assembly system, module or interface 12 maybe used to specify a first set of treatment recommendations or outcomeshaving a first set of associated treatment confidence metric values.However, the evaluation and prioritization of the patient attributes(and associated treatment confidence metric values) at the treatmentrecommendation system, module or interface 13 can result in a second,different set of patient attributes and/or treatment recommendationsbeing displayed for selection and input during a second pass through theassembly system, module or interface 12, where the set of patientattributes are ranked or prioritized on the basis of which attributeswill most effectively increase the confidence for the treatmentrecommendations. For example, a first set of ranked patient attributes(e.g., test results for hemoglobin A1 c, bronchoscopy, blood creatinine,parathyroid hormones, foot numbness, polysomnography, urinalysis) andassociated treatment recommendations that are generated for displayduring an initial pass through the assembly, system, module or interface12 may be evaluated against additional attribute input data whengenerating new treatment recommendations with the treatmentrecommendation system, module or interface 13, resulting in a second setof ranked patient attributes (e.g., test results for blood creatinine,magnetic resonance, bronchoscopy, parathyroid hormones, foot numbness,polysomnography, urinalysis) that may be generated for display withassociated treatment recommendations. In response, the physician ormedical expert can select and update one of the second set of rankedpatient attributes with additional patient attribute data, and thesequence repeats until a desired level of confidence is obtained for thetreatment recommendations.

Based on interactions with the patient attribute inputs, the evaluatedtreatment recommendations may be displayed by the iterative attributeacquisition engine 11 as an optimized visualization of patientattributes and treatment recommendations for the patient. To this end,the iterative attribute acquisition engine 11 may include avisualization optimizer 15 that is configured to display an optimizedvisualization of the prioritized patient attributes and treatmentrecommendations for the patient. The optimized visualization generatedby the visualization optimizer 15 may specify the content and style forconveying comparative patient attributes and treatment options, such asby generating a patient case matrix interface with ranked patientattributes presented in a first column and with treatment optionspresented in a second column. Next to each ranked patient attribute, thepatient case matrix interface may include a confidence improvementmetric indicator which visually represents the confidence uncertaintymetric that can be reduced by providing additional patient attributedata. In addition, the patient case matrix interface may include apatient care customization metric indicator for each patient whichvisually represents what percentage of the patient's attributes havebeen assembled, effectively representing the amount of uncertaintyassociated with the treatment recommendations for that patient. Thedisplayed patient care customization metric indicator may also include amarginal customer care confidence uncertainty metric indicator thatvisually represents how much uncertainty can be reduced by providingadditional patient attribute data.

Types of information handling systems that can utilize QA system 1 rangefrom small handheld devices, such as handheld computer/mobile telephone120 to large mainframe systems, such as mainframe computer 170. Examplesof handheld computer 120 include personal digital assistants (PDAs),personal entertainment devices, such as MP3 players, portabletelevisions, and compact disc players. Other examples of informationhandling systems include pen, or tablet, computer 130, laptop, ornotebook, computer 140, personal computer system 150, and server 160. Asshown, the various information handling systems can be networkedtogether using computer network 9. Types of computer network 9 that canbe used to interconnect the various information handling systems includeLocal Area Networks (LANs), Wireless Local Area Networks (WLANs), theInternet, the Public Switched Telephone Network (PSTN), other wirelessnetworks, and any other network topology that can be used tointerconnect the information handling systems. Many of the informationhandling systems include nonvolatile data stores, such as hard drivesand/or nonvolatile memory. Some of the information handling systems mayuse separate nonvolatile data stores (e.g., server 160 utilizesnonvolatile data store 165, and mainframe computer 170 utilizesnonvolatile data store 175). The nonvolatile data store can be acomponent that is external to the various information handling systemsor can be internal to one of the information handling systems. Anillustrative example of an information handling system showing anexemplary processor and various components commonly accessed by theprocessor is shown in FIG. 2.

FIG. 2 illustrates information handling system 200, more particularly, aprocessor and common components, which is a simplified example of acomputer system capable of performing the computing operations describedherein. Information handling system 200 includes one or more processors210 coupled to processor interface bus 212. Processor interface bus 212connects processors 210 to Northbridge 215, which is also known as theMemory Controller Hub (MCH). Northbridge 215 connects to system memory220 and provides a means for processor(s) 210 to access the systemmemory. In the system memory 220, a variety of programs may be stored inone or more memory device, including an optimized iterative attributeacquisition engine module 221 which may be invoked to dynamicallyoptimize the acquisition of query attributes that would significantlyimprove the QA process by using cognitive computer assisted iterativedisclosure techniques to identify query attributes that are ranked bythe number of possible conclusions that can be deprioritized by eachquery attribute. Graphics controller 225 also connects to Northbridge215. In one embodiment, PCI Express bus 218 connects Northbridge 215 tographics controller 225. Graphics controller 225 connects to displaydevice 230, such as a computer monitor.

Northbridge 215 and Southbridge 235 connect to each other using bus 219.In one embodiment, the bus is a Direct Media Interface (DMI) bus thattransfers data at high speeds in each direction between Northbridge 215and Southbridge 235. In another embodiment, a Peripheral ComponentInterconnect (PCI) bus connects the Northbridge and the Southbridge.Southbridge 235, also known as the I/O Controller Hub (lap is a chipthat generally implements capabilities that operate at slower speedsthan the capabilities provided by the Northbridge. Southbridge 235typically provides various busses used to connect various components.These busses include, for example, PCI and PCI Express busses, an ISAbus, a System Management Bus (SMBus or SMB), and/or a Low Pin Count(LPC) bus. The LPC bus often connects low-bandwidth devices, such asboot ROM 296 and “legacy” I/O devices (using a “super I/O” chip). The“legacy” I/O devices (298) can include, for example, serial and parallelports, keyboard, mouse, and/or a floppy disk controller. Othercomponents often included in Southbridge 235 include a Direct MemoryAccess (DMA) controller, a Programmable Interrupt Controller (PIC), anda storage device controller, which connects Southbridge 235 tononvolatile storage device 285, such as a hard disk drive, using bus284.

ExpressCard 255 is a slot that connects hot-pluggable devices to theinformation handling system. ExpressCard 255 supports both PCI Expressand USB connectivity as it connects to Southbridge 235 using both theUniversal Serial Bus (USB) the PCI Express bus. Southbridge 235 includesUSB Controller 240 that provides USB connectivity to devices thatconnect to the USB. These devices include webcam (camera) 250, infrared(IR) receiver 248, keyboard and trackpad 244, and Bluetooth device 246,which provides for wireless personal area networks (PANs). USBController 240 also provides USB connectivity to other miscellaneous USBconnected devices 242, such as a mouse, removable nonvolatile storagedevice 245, modems, network cards, ISDN connectors, fax, printers, USBhubs, and many other types of USB connected devices. While removablenonvolatile storage device 245 is shown as a USB-connected device,removable nonvolatile storage device 245 could be connected using adifferent interface, such as a Firewire interface, etc.

Wireless Local Area Network (LAN) device 275 connects to Southbridge 235via the PCI or PCI Express bus 272. LAN device 275 typically implementsone of the IEEE 802.11 standards for over-the-air modulation techniquesto wireless communicate between information handling system 200 andanother computer system or device. Extensible Firmware Interface (EFI)manager 280 connects to Southbridge 235 via Serial Peripheral Interface(SPI) bus 278 and is used to interface between an operating system andplatform firmware. Optical storage device 290 connects to Southbridge235 using Serial ATA (SATA) bus 288. Serial ATA adapters and devicescommunicate over a high-speed serial link. The Serial ATA bus alsoconnects Southbridge 235 to other forms of storage devices, such as harddisk drives. Audio circuitry 260, such as a sound card, connects toSouthbridge 235 via bus 258. Audio circuitry 260 also providesfunctionality such as audio line-in and optical digital audio in port262, optical digital output and headphone jack 264, internal speakers266, and internal microphone 268. Ethernet controller 270 connects toSouthbridge 235 using a bus, such as the PCI or PCI Express bus.Ethernet controller 270 connects information handling system 200 to acomputer network, such as a Local Area Network (LAN), the Internet, andother public and private computer networks.

While FIG. 2 shows one information handling system, an informationhandling system may take many forms, some of which are shown in FIG. 1.For example, an information handling system may take the form of adesktop, server, portable, laptop, notebook, or other form factorcomputer or data processing system. In addition, an information handlingsystem may take other form factors such as a personal digital assistant(PDA), a gaming device, ATM machine, a portable telephone device, acommunication device or other devices that include a processor andmemory. In addition, an information handling system need not necessarilyembody the north bridge/south bridge controller architecture, as it willbe appreciated that other architectures may also be employed.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 3which illustrates a simplified block diagram form chart 30 showing thelogic for using a machine learning process to iteratively acquire queryattributes and generate a list of ranked answers. As disclosed herein,the iterative acquisition of query attributes and the generation ofranked query attributes and answers therefrom may be performed by acognitive system, such as the QA system 1 or any suitable informationhandling system.

As an initial step in the process 30 after the method starts (step 31),query attributes are received at step 32. In selected embodiments, thereceived query attributes may be attributes relating to the medicalcondition of a patient that are assembled from patient profile data(e.g., age and gender), clinical data, diagnostic data, medical data(e.g., referral requests, patient allergies, prescription renewals, labreports, and other patient health data), and/or any health-related data;including but not limited to patient medical records, patient enteredinformation, care team entered information, healthcare device generatedinformation, billing information, etc. In other embodiments, the queryattributes relate to questions in other fields besides medicine, such asfinance, scientific research, engineering, software, or the like. Inselected medical fields of application, the displayed query attributesmay be patient attributes that are displayed for comparison andselection in a patient case matrix interface in which ranked patientattributes are listed in a first column and patient treatmentrecommendations are listed in a second column. As disclosed herein, thepatient attributes may be ranked by a confidence improvement metricwhich quantities the confidence uncertainty that can be reduced byproviding additional patient attribute data. In addition or in thealternative, each displayed patient attribute may include a confidenceimprovement metric indicator which visually represents the confidenceuncertainty metric that can be reduced by providing additional patientattribute data. In addition or the alternative; the displayed queryattributes may include a patient care customization metric indicator foreach patient which visually represents what percentage of the patient'sattributes have been assembled, effectively representing the amount ofuncertainty associated with the treatment recommendations for thatpatient. The displayed patient care customization metric indicator mayalso include a marginal customer care confidence uncertainty metricindicator that visually represents how much uncertainty can be reducedby providing additional patient attribute data. Through user interactionwith the displayed query attributes; the processing at step 32 may alsoinclude the acquisition of one or more query attribute values throughuser interaction, such as the selection and/or input of query attributedata by a user (e.g., a physician, medical assistance; patient orexpert; who submits query interaction data or values for a selectedquery attribute). In whole or in part, the acquisition of queryattributes may be automatically or programmatically implemented byretrieving data from memory storage. In selected embodiments, queryinteraction data may be patient attribute data, such as information ortest result values that are entered for a selected patient attribute.The acquisition processing at step 32 may include the generation anddisplay of a query attribute input field which the user may use to inputpatient attribute data or information, such as a diagnostic or medicaltest result value.

At step 33, a natural language processing (NLP) technique, such asmachine learning and/or deep analytic analysis, may be applied to thequery attributes, including any newly entered query attribute values, togenerate answers or query responses along with associated confidencemeasures based on the current query attributes. In selected exampleembodiments, NLP-based machine learning techniques applied at step 33may invoke association rules and/or pattern recognition logic whichprocess patient attribute data, including acquired patient attributevalues, to generate treatment recommendations having associatedtreatment confidence metric values for the current patient case.

At step 34, an NLP technique, such as machine learning and/or deepanalytic analysis, may be applied to rank or re-rank the missing queryattributes. In selected example embodiments, the NLP-based machinelearning techniques applied at step 34 may apply machine learningtechniques to calculate a confidence improvement metric value for eachpatient attribute which quantifies the confidence uncertainty that canbe reduced by providing additional data or information for that patientattribute. Using the calculated confidence improvement metric values,the patient attributes can be ranked at step 34 to prioritize thepatient attributes in terms of the marginal reduction in uncertaintythat can be achieved by entering information to improve the confidencevalues for a set of treatment recommendations or options. In selectedembodiments, the applied machine learning techniques may prioritize thepatient attributes on the basis of a confidence hyper-value metric(e.g., an area, volume, or other multi-dimensional value, depending onthe number of attributes), alone or in combination with a normalizedrepresentation of the confidence hyper-value metric, to measure theuncertainty in the confidence values for the options provided. Asdisclosed herein, the confidence hyper-value metric may be computed asthe product of option confidence ranges for each patient attribute orpiece of information. The patient attributes or information can then beranked based on a marginal reduction in the confidence volume.

At step 35, the answers and associated confidence measures may bedisplayed for user viewing and interaction. As displayed, the answersare ordered or ranked by the associated confidence measure. For example,when a “treatment” tab on a user interface is selected by usermanipulation of a screen cursor, the user interface may then display aplurality of treatment “answers” ranked by associated confidencemeasures.

At step 36, the missing query attributes may be displayed for userviewing and interaction, where the query attributes are ranked orordered by each query attribute's impact to improve the answerconfidence. For example, when a “investigation” tab on a user interfaceis selected by user manipulation of a screen cursor, the user interfacemay then display a plurality of missing query attributes that areordered or ranked by the marginal reduction in uncertainty that can beachieved by entering information to improve the confidence values for aset of treatment recommendations or options (e.g., based on the marginalreduction in the confidence volume).

As indicated with the feedback path 38, the sequence of steps 32-36 canbe repeated in an iterative loop for so long as the query process isongoing (e.g., negative outcome to decision step 37). In this way, thecomputed confidence improvement metric values may be displayed with theranked query attributes for use in allowing users to optimally improvetheir queries through a guided iterative approach. This capability isreferred to as “iterative disclosure.” However, once the query processis completed (e.g., affirmative outcome to decision step 37), theprocess 30 is done (step 39).

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, there is now disclosed anddescribed a machine learning process to optimally improve the submissionof queries to a cognitive system, such as the QA system 1 or anysuitable information handling system, through a guided iterativeapproach which uses answer confidence values to identify and rank queryattributes that should be submitted to reduce the confidence uncertaintyfor the answers. While described hereinbelow in the context of a healthcare treatment advisor which uses treatment confidence values tocalculate a “Customer Care Measure” figure of merit, it will beappreciated that the present disclosure may be applied to a variety ofdifferent applications or fields. In the disclosed machine learningprocess, there is a fixed and finite set of query or patient attributesS_(A)={A₁, A₂, A₃ . . . A_(m)} and a fixed and finite set of answer ortreatment outcomes S_(T)={T₁, T₂, T₃ . . . T_(n)}. In this setting, aspecific patient, P_(i), may be characterized with a vectorrepresentation of the patient's attributes a=[a₁, a₂, a₃ . . . a_(m)](when all in attributes are known) or with a vector representationa′=[a₁, a₂, a₃ . . . a_(k), u_(k+1), u_(k+2) . . . u_(m)] (when only kattributes are known and the u_(i) attribute values are not known). Inaddition, the answer or treatment outcomes for the specific patientP_(i) may be characterized with a vector representation of theconfidence values associated with each treatment t=[t₁, t₂, t₃ . . .t_(n)] for a given attribute vector a. Using these vectorrepresentations, the cognitive system maps each possible patientattribute vector to a treatment confidence vector space in order todetermine the hyper-value metric (e.g., space or volume) encompassingthe known patient attributes as a representation of the total confidenceuncertainty measure for the set of answer or treatment outcomes.

Turning now to FIGS. 4-6, there is provided a simplified example of thedisclosed machine learning process for using answer confidence values toidentify and rank query attributes that should be submitted to reducethe confidence uncertainty for the answer or treatment outcomes. In thissimplified example, there are only two possible query attributes (e.g.,patient attributes) a1, a2, each of which may have three values: low,medium, and high. It is assumed further that there are only two possibleoutcomes or answers (e.g., treatments) t1, t2. Representing the queryattributes as an attribute vector, the cognitive system maps eachpossible patient attribute vector to a treatment confidence vector spacein which each vector in the treatment confidence space represents thecalculated confidence measure for each treatment. In FIG. 4, the firstdiagram 41 represents the attribute vector 43 in which all patientattributes are known with the value for the first attribute a1 (e.g.,bronchoscopy) being “medium” and the value for the second attribute a2(e.g., blood creatinine) being “low.” In an assumed case where there areonly two possible answers or treatments, t1 (e.g., low-dose aspirin) andt2 (e.g., Warfarin), the cognitive system maps each possible patientattribute vector 41 to a treatment confidence vector space 42 whereineach vector in the treatment confidence space represents the treatmentconfidence that is calculated by the cognitive system. In the example ofFIG. 4 where all patient attributes are known, the cognitive systemprovides a single confidence 44 for each treatment option, resulting ina confidence range spread of zero (e.g., “no spread”) for each treatmentto indicate a mid-range confidence for the first treatment (e.g.,low-dose aspirin (t1)) and a slightly lower confidence for the secondtreatment (e.g., Warfarin (t2)).

Referring now to FIG. 5, there is shown a second mapping of attributevectors 51 to a confidence vector space 52 in a case where none of theattribute vector values are known. In particular, the first diagram 51represents the attribute vector 53 in which none of the patientattributes are known, such that the values for the first attribute a1(e.g., bronchoscopy) and the second attribute a2 (e.g., bloodcreatinine) could be “low,” “medium” or “high.” In this case where eachpossible patient attribute vector 51 is mapped to a treatment confidencevector space 52, the cognitive system is unable to choose a specifictreatment confidence vector, and instead a range or spread of confidencevalues is determined for each treatment, including a first treatmentrange (e.g., t1 spread) and a second treatment range (e.g., t2 spread).By multiplying the spread in confidence for each treatment, thecognitive system may calculate the overall confidence uncertainty as ahyper-value. In the case of a two-dimensional treatment confidence space52 where nothing is known about a patient attributes a1, a2, a nominalconfidence uncertainty measure hyper-value is computed as a nominalvolume (e.g., Vnominal) 54, where the term “volume” is used to representconfidence uncertainty for all possible dimensionalities.

Referring now to FIG. 6, there is shown a third mapping of attributevectors 61 to a confidence vector space 62 where one or more of theattribute vector values are known and others are unknown. In particular,the first diagram 61 represents the attribute vector 63 in which one ofthe patient attributes are known such that the first attribute a1 (e.g.,bronchoscopy) has a known “medium” value, but the second attribute a2(e.g., blood creatinine) is not known. In this case where each possiblepatient attribute vector 61 is mapped to a treatment confidence vectorspace 62, the confidence uncertainty decreases and the cognitive systemis able to calculate an associated uncertainty volume for the knownpatient attributes by multiplying the spread in confidence for eachtreatment range (e.g., t1 spread and t2 spread) to determine the overallconfidence uncertainty as a known hyper-value (e.g., Vknown) 64. Theuncertainty volume associated with a known set of attributes and anotherunknown attribute, A, is V_(A). This volume will be less than or equalto Vknown—the volume of the known attributes.

As will be appreciated, the simplified example described with referenceto FIGS. 4-6 can be extended to the case where there are m patientattributes in which case the number of patient attribute vectors will bethe product of the number of possible values for each attribute, e.g.count(a)=count(a₁)*count(a₂)*count(a₃) . . . count(a_(m)). Even for amodest 10 attributes with three levels each, this would result in3¹⁰=59,049 possible patient attribute vectors. For this reason, themapping of patient attribute vectors to treatment confidence vectors(which involves expensive QA system pipeline calculations) should beexecuted in batches and the results should be stored for quick retrievalduring application use. This is possible since the mapping will notchange unless the corpus changes.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIGS.7-11 which illustrate a sequence of user interface display screens 70,80, 90, 100, 110 to illustrate the iterative acquisition of patientattributes and resulting treatment recommendations based on computedconfidence volume, customer care metric (CCM) and marginal CCM valueswhich are used to optimally rank and display the patient attributes andtreatment options presented to the user. As disclosed herein, thegeneration and display of the user interface display screens and theprocessing of user interactions therewith may be performed by acognitive system, such as the QA system 1 or any suitable informationhandling system.

Referring first to FIG. 7, there is shown a first user interface displayscreen 70 which lists one or more patient cases 71 with suitabledescriptions 71 a-h for each patient which are configured to be selectedin response to user interaction, such as by using a screen touch orcursor 73. For each listed patient, the user interface display screen 70may also display a patient care customization metric indicator 72 foreach patient, where each indicator 72 a-h visually represents whatpercentage of the patient's attributes have been assembled, effectivelyrepresenting the amount of uncertainty associated with the treatmentrecommendations for that patient. Through user interaction with thetouch screen or cursor, one of the patient cases (e.g., 71 g) may beselected.

Referring now to FIG. 8, there is shown a second user interface displayscreen 80 after selection of the patient case 71 g shown in FIG. 7. Asillustrated, the second user interface display screen 80 may include adescription panel 81 for the selected patient, the corresponding patientcare customization metric indicator 82, an investigation screen tab 83,and a treatments screen tab 84.

The patient description panel 81 may display patient information, suchas demographic and medical data for the selected patient case. Forexample, the depicted patient description panel 81 identifies theselected patient's medical data as “Atrial fibrillation, CHF,Hypertension,” and identifies the selected patient's demographic data as“Age: 69 Gender: Male Stage: C—Symptomatic.”

The patient care customization metric indicator 82 may include acustomer care measure (CCM) 88 which visually represents what percentageof the patient's attributes have been assembled, effectivelyrepresenting the amount of uncertainty associated with the treatmentrecommendations for that patient. In selected embodiments, the currentcustomer care measure (CCM) may be calculated from confidence volumevalues as CCM=1−(Vknown/Vempty), where Vknown is the confidence volumefor a set of known attributes (represented by the vector a′) and whereVempty is the confidence volume when no patient attributes are known.The patient care customization metric indicator 82 may also include anMCCM indicator 89 of the marginal improvement in the customer caremeasure that visually represents how much uncertainty can be reduced byproviding additional patient attribute data for a selected patientattribute. In selected embodiments, the marginal MCCM increase for anattribute A may be calculated from confidence volume values asMCCM=(V_(A)−Vknown)/Vempty, where V_(A) is the confidence volume for aset of known attributes and one additional attribute A, where Vknown isthe confidence volume for a set of known attributes (represented by thevector a′), and where Vempty is the confidence volume when no patientattributes are known.

When selected, the investigation screen tab 83 lists a plurality ofpatient attributes 85 for the selected patient case which are configuredto be selected in response to user interaction, such as by using a touchor screen cursor 73, with each selected patient attribute including apanel description of the patient attribute, such as an overview andreview of potentially impacted treatments. Next to each listed patientattribute 85 a-g is a corresponding improvement measure 86 a-g whichidentifies or quantifies the available improvement in treatmentconfidence that can be achieved by specifying additional input valuesfor the patient attribute. Computed as a hyper-value metric product oftreatment confidence ranges encompassing the patient attributes 85 a-g,the improvement measures 86 a-g quantify a marginal reduction in theconfidence uncertainty measures, and may be used to rank the display ofpatient attributes 85 a-g on the basis of which additional attributeinput data will most effectively increase the confidence for thetreatment recommendations. The data values for each patient attribute 85a-g may have a continuous or categorical and fixed value associatedtherewith that is either stored in memory or input by the user. To allowuser input of patient attribute values, the second user interfacedisplay screen 80 may be configured to respond to user interactionselection of a patient attribute (e.g., Hemoglobin Ale 85 a) to displayan attribute input field 87 (e.g., Hemoglobin Ale input value) that ispresented for user input of data relating to the selected patientparameter.

When the treatments screen tab 84 is selected, the user interfacedisplay screen 80 may list a plurality of treatment recommendations(e.g., “low-dose aspirin,” “Warfarin,” Metoprolol”) for the selectedpatient case which are generated from the patient attribute data. Inaddition, treatment details may be displayed for each selected treatmentrecommendation, including supporting evidence for treatment, coreperformance measures, etc. Next to each treatment recommendation, acorresponding treatment confidence measure indicator may be displayedwhich identifies the confidence calculated for the treatment by thecognitive system.

In general terms, the patient attributes 85 a-g displayed in theinvestigation screen tab 81 are selected from a fixed and finite set ofpatient attributes S_(A)={A₁, A₂, A₃ . . . A_(m)}. In addition, thespecific attributes 85 for a selected patient P; may be represented invector form as attribute vector a=[a₁, a₂, a₃ . . . a_(m)], while theattribute vector a′=[a₁, a₂, a₃ . . . a_(k), u_(k+1), u_(k+2) . . .u_(m)] may be used to provide a vector representation of a specificpatient's attributes when one or more attributes are unknown, where thevalues u_(i) are null. In addition, the treatment confidence measuresmay be represented in vector form as a treatment confidence vectort=[t₁, t₂, t₃ . . . t_(n)] for a given attribute vector, a. Eachtreatment confidence value may have a continuous value ranging from 0 to1 inclusive that is either stored in memory or generated by thecognitive system.

Referring now to FIG. 9, there is shown a third user interface displayscreen 90 after input of patient attribute data 91 at the input field87. As illustrated, the third user interface display screen 90 includesthe same patient case description panel 81, patient care customizationmetric indicator 82, investigation screen tab 83, and treatments screentab 84. As shown at this stage, the ranking and improvement outcomes forthe listed patient attributes 85, 86 are not changed until after theadditional patient attribute input data is entered in response to userinteraction, such as by using a screen touch or cursor 73 to activate a“submit” button 92.

Referring now to FIG. 10, there is shown a fourth user interface displayscreen 100 after submission of the additional patient attribute inputdata as shown in FIG. 9. As illustrated, the fourth user interfacedisplay screen 100 includes the same description panel 81 for theselected patient and patient care customization metric indicator 82, butnow updated with a new CCM indicator 108 which reflects the reducedamount of uncertainty associated with the treatment recommendations forthat patient after submission of the patient attribute input data. Inaddition, the patient care customization metric indicator 82 is updatedwith a new MCCM indicator 109 to reflect how much uncertainty can bereduced by providing additional patient attribute data for a newlyselected or top-ranked patient attribute (e.g., patient attribute 105a). In addition, the investigation screen tab 83 and treatments screentab 84 in the fourth user interface display screen 100 are updated toreflect a new ranking of patient attributes 105, improvement measures106, and recommended treatments (not shown). For example, in the newranking of patient attributes 105 a-g are shown with correspondingimprovement measures 106 a-g which are sorted or ranked to identifywhich additional attribute input data will most effectively increase theconfidence for the treatment recommendations. Under the new ranking, thelisting of patient attributes 105 a-g (e.g., blood creatinine, magneticresonance, bronchoscopy, parathyroid hormones, foot numbness,polysomnography, urinalysis) replaces the listing of patient attributes85 a-g (e.g., hemoglobin A1c, bronchoscopy, blood creatinine,parathyroid hormones, foot numbness, polysomnography, urinalysis). As aresult of the new ranking of patient attributes 105, the investigationscreen tab 83 on the fourth user interface display screen 100 mayinclude a new patient attribute input field 107 and panel description ofthe selected or top ranked patient attribute (e.g., 105 a) which mayinclude an overview and review of potentially impacted treatments forthis patient attribute (e.g., blood creatinine).

Referring now to FIG. 11, there is shown a fifth user interface displayscreen 110 after the treatments screen tab 84 is selected, such as byusing a touch or screen cursor 73 as shown in FIG. 10. As illustrated,the fifth user interface display screen 110 includes a list of treatmentrecommendations 115 (e.g., “low-dose aspirin,” “Warfarin,” Metoprolol,”etc.) for the selected patient case which are generated by the cognitivesystem from the patient attribute data. In addition, treatment detailsfor a selected treatment recommendation may be displayed in a displaypanel 114, including an overview description with supporting evidencefor treatment, core performance measures, etc. Next to each treatmentrecommendation 115 a-g, a corresponding treatment confidence measureindicator 116 a-g may be displayed which identifies the confidencecalculated for the treatment by the cognitive system. While anyindicator icon may be used, selected embodiments of the treatmentconfidence measure indicator 116 a-g employ a completion ring iconvisually represents the confidence measure as a percentage of ring thatis colored or shaded to represent the amount of confidence associatedwith the treatment recommendation.

Selected embodiments of the present disclosure are described withreference to a QA system for displaying optimized treatmentrecommendations and outcomes on the basis of patient-specified inputparameters (e.g., patient profile and medical data, patient-specifiedattributes, importance and impact factors, etc.), though otherinformation handling systems or computing devices may be used. Asdisclosed and described herein, the processing implemented by thecognitive system, such as a QA system 1, is operative to update andguide the acquisition of query attributes whenever the corpus isingested or updated with new query attributes, at which point thecognitive system maps each possible attribute vector, a, to a singletreatment confidence vector, t. The number of treatments in S_(T) maydiffer from the number of confidence vectors, t. For any patient with aset of known and unknown patient attributes, a′, the cognitive systemwill calculate and current CCM for display on the user interface displayscreen. For each unknown attribute, the cognitive system will calculatethe marginal CCM for display on the user interface display screen andfor use in ranking the patient attributes displayed thereon to guide aniterative attribute input process. However implemented, it will beappreciated that the present disclosure may employ a user interface togenerate and display personalized treatment recommendations withassociated treatment outcomes for visual comparison by iterativelyguiding the patient attribute input process to improve the treatmentrecommendations by identifying which patient attributes should beupdated to most effectively reduce the uncertainty in the confidence forthe treatment recommendations.

By now, it will be appreciated that there is disclosed herein a system,method, apparatus, and computer program product for displaying queryitems to a user at an information handling system having a processor anda memory. As disclosed, the system, method, apparatus, and computerprogram product generate a plurality answer items (e.g., medicaltreatment recommendations for a selected patient) having a correspondingplurality of confidence values from a plurality of query items (e.g.,medical attributes for the selected patient) comprising one or morespecified query items and one or more unspecified query items. Inaddition, the information handling system performs a ranking analysis ofquery items by computing a current answer confidence metric for thespecified query items and a marginal answer confidence improvementmetric for each unspecified query item in the first plurality of queryitems. In selected embodiments, the ranking analysis may be performed bygenerating a first vector representation a=[a₁, a₂, a₃ . . . a_(m)] of mspecified query items; generating a second vector representation a′=[a₁,a₂, a₃ . . . a_(k), u_(k+1), u_(k+2) . . . u_(m)] of the plurality ofquery items comprising k specified query items and unspecified queryitems u_(i); mapping the first and second vector representations tovector representations of the confidence values for each answer item;and calculating confidence hyper-value metrics from the vectorrepresentations of the confidence values to compute the current answerconfidence metric and the marginal answer confidence improvement metric.In other embodiments, the ranking analysis may be performed by running acognitive analysis comparison of each marginal answer confidenceimprovement metric for each unspecified query item in the firstplurality of query items to rank the query items in sorted order fromlargest to smallest marginal answer confidence improvement metric. Oncethe query items are ranked, the information handling system may displayone or more unspecified query items in sorted order based on themarginal answer confidence improvement metric for each unspecified queryitem. In displaying the query items; the information handling system mayprioritize a first query item having a largest marginal answerconfidence improvement metric. Once the query items are displayed, theinformation handling system may obtain user-specified attribute data forat least one of the first plurality query items, and then perform asecond ranking analysis of query items by computing an updated currentanswer confidence metric for the specified query items and an updatedmarginal answer confidence improvement metric for each unspecified queryitem in the first plurality of query items, and then display theunspecified query item(s) in sorted order based on the updated marginalanswer confidence improvement metric for each unspecified query item. Inselected embodiments, the steps of performing the ranking analysis anddisplaying the one or more unspecified query items in sorted order arerepeated in response to additional query attribute data for at least oneof the first plurality query items to provide an iterative guidedapproach for generating answer items.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, changes and modifications may be madewithout departing from this invention and its broader aspects,Therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention. Furthermore, it is to be understood that theinvention is solely defined by the appended claims. It will beunderstood by those with skill in the art that if a specific number ofan introduced claim element is intended, such intent will be explicitlyrecited in the claim, and in the absence of such recitation no suchlimitation is present. For non-limiting example, as an aid tounderstanding, the following appended claims contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimelements. However, the use of such phrases should not be construed toimply that the introduction of a claim element by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim element to inventions containing only one such element,even when the same claim includes the introductory phrases “one or more”or “at least one” and indefinite articles such as “a” or “an”; the sameholds true for the use in the claims of definite articles.

What is claimed is:
 1. A method, in an information handling systemcomprising a processor and a memory, for displaying query items, themethod comprising: performing, at the information handling system, aranking analysis of the plurality of query items by computing a currentanswer confidence metric for the specified query items and a marginalanswer confidence improvement metric quantifying how much answeruncertainty can be reduced by providing additional attribute data foreach unspecified query item in the plurality of query items; anddisplaying in ranked order based on the marginal answer confidenceimprovement metric for each unspecified query item, the one or moreunspecified query items along with corresponding marginal answerconfidence improvement metrics to visually show how much uncertainty canbe reduced by providing additional attribute data for each unspecifiedquery item.
 2. The method of claim 1, where the plurality of query itemscomprises medical attributes for a selected patient and where theplurality of answer items comprises medical treatment recommendationsfor the selected patient that are generated from the medical attributesfor the selected patient.
 3. The method of claim 1, where performing theranking analysis comprises: generating, by the information handlingsystem, a first vector representation a=[a₁, a₂, a₃ . . . a_(m)] of mspecified query items; generating, by the information handling system, asecond vector representation a′=[a₁, a₂, a₃ . . . a_(k), u_(k+1),u_(k+2) . . . u_(m)] of the plurality of query items comprising kspecified query items and i unspecified query items, where k+i=m;mapping, by the information handling system, the first and second vectorrepresentations to vector representations of the confidence values foreach answer item; and calculating, by the information handling system,confidence hyper-value metrics from the vector representations of theconfidence values to compute the current answer confidence metric andthe marginal answer confidence improvement metric.
 4. The method ofclaim 1, where displaying the one or more unspecified query items inranked order comprises prioritizing a first query item having a largestmarginal answer confidence improvement metric.
 5. The method of claim 1,further comprising obtaining, by the information handling system,user-specified attribute data for at least one of the first query items.6. The method of claim 5, further comprising: performing, at theinformation handling system, a second ranking analysis of query itemsafter obtaining the user-specified attribute data by computing anupdated current answer confidence metric for the specified query itemsand an updated marginal answer confidence improvement metric for eachunspecified query item in the plurality of query items; and displaying,by the information handling system, one or more unspecified query itemsin ranked order based on the updated marginal answer confidenceimprovement metric for each unspecified query item.
 7. The method ofclaim 1, further comprising repeating the steps of performing theranking analysis and displaying the one or more unspecified query itemsin ranked order in response to additional query attribute data for atleast one of the plurality query items to provide an iterative guidedapproach for generating answer items.
 8. The method of claim 1, whereperforming the ranking analysis comprises running a cognitive analysiscomparison of each marginal answer confidence improvement metric foreach unspecified query item in the plurality of query items to rank thequery items in ranked order from largest to smallest marginal answerconfidence improvement metric.
 9. An information handling systemcomprising: one or more processors; a memory coupled to at least one ofthe processors; a set of instructions stored in the memory and executedby at least one of the processors to display query items, wherein theset of instructions are executable to perform actions of: performing, atthe system, a ranking analysis of the plurality of query items bycomputing a current answer confidence metric for the specified queryitems and a marginal answer confidence improvement metric quantifyinghow much answer uncertainty can be reduced by providing additionalattribute data for each unspecified query item in the plurality of queryitems; and displaying, in ranked order based on the marginal answerconfidence improvement metric for each unspecified query item, the oneor more unspecified query items along with corresponding marginal answerconfidence improvement metrics to visually show how much uncertainty canbe reduced by providing additional attribute data for each unspecifiedquery item.
 10. The information handling system of claim 9, where theplurality of query items comprises medical attributes for a selectedpatient and where the plurality of answer items comprises medicaltreatment recommendations for the selected patient that are generatedfrom the medical attributes for the selected patient.
 11. Theinformation handling system of claim 9, where wherein the set ofinstructions are executable to perform the ranking analysis by:generating a first vector representation a=[a₁, a₂, a₃ . . . a_(m)] of mspecified query items; generating a second vector representation a′=[a₁,a₂, a₃ . . . a_(k), u_(k+1), u_(k+2) . . . u_(m)] of the plurality ofquery items comprising k specified query items and unspecified queryitems u_(i); mapping the first and second vector representations tovector representations of the confidence values for each answer item;and calculating confidence hyper-value metrics from the vectorrepresentations of the confidence values to compute the current answerconfidence metric and the marginal answer confidence.
 12. Theinformation handling system of claim 9, wherein the set of instructionsare executable to display the one or more unspecified query items inranked order by prioritizing a first query item having a largestmarginal answer confidence improvement metric.
 13. The informationhandling system of claim 9, further comprising a set of instructionsthat are executable to perform actions of obtaining user-specifiedattribute data for at least one of the plurality query items.
 14. Theinformation handling system of claim 13, further comprising a set ofinstructions that are executable to perform actions of: performing asecond ranking analysis of query items after obtaining theuser-specified attribute data by computing an updated current answerconfidence metric for the specified query items and an updated marginalanswer confidence improvement metric for each unspecified query item inthe plurality of query items; and displaying one or more unspecifiedquery items in ranked order based on the updated marginal answerconfidence improvement metric for each unspecified query item.
 15. Theinformation handling system of claim 9, further comprising a set ofinstructions that are executable to repeat the steps of performing theranking analysis and displaying the one or more unspecified query itemsin ranked order in response to additional query attribute data for atleast one of the plurality query items to provide an iterative guidedapproach for generating answer items.
 16. The information handlingsystem of claim 9, wherein the set of instructions are executable toperform the ranking analysis by running a cognitive analysis comparisonof each marginal answer confidence improvement metric for eachunspecified query item in the plurality of query items to rank the queryitems in ranked order from largest to smallest marginal answerconfidence improvement metric.
 17. A computer program product comprisinga computer readable storage medium having computer instructions storedtherein that, when executed by an information handling system, cause thesystem to display query items by: performing, at the system, a rankinganalysis of the plurality of query items by computing a current answerconfidence metric for the specified query items and a marginal answerconfidence improvement metric quantifying how much answer uncertaintycan be reduced by providing additional attribute data for eachunspecified query item in the plurality of query items; and displaying,in ranked order based on the marginal answer confidence improvementmetric for each unspecified query item, the one or more unspecifiedquery items along with corresponding marginal answer confidenceimprovement metrics to visually show how much uncertainty can be reducedby providing additional attribute data for each unspecified query item.18. The computer program product of claim 17, where the plurality ofquery items comprises medical attributes for a selected patient andwhere the plurality of answer items comprises medical treatmentrecommendations for the selected patient that are generated from themedical attributes for the selected patient.
 19. The computer programproduct of claim 17, where performing the ranking analysis comprises:generating a first vector representation a=[a₁, a₂, a₃ . . . a_(m)] of mspecified query items; generating a second vector representation a′=[a₁,a₂, a₃ . . . a_(k), u_(k+1), u_(k+2) . . . u_(m)] of the plurality ofquery items comprising k specified query items and unspecified queryitems u_(i); mapping the first and second vector representations tovector representations of the confidence values for each answer item;and calculating confidence hyper-value metrics from the vectorrepresentations of the confidence values to compute the current answerconfidence metric and the marginal answer confidence improvement metric.20. The computer program product of claim 17, where displaying the oneor more unspecified query items in ranked order comprises prioritizing afirst query item having a largest marginal answer confidence improvementmetric.
 21. The computer program product of claim 17, further comprisingcomputer instructions that, when executed by the information handlingsystem, cause the system to obtain user-specified attribute data for atleast one of the plurality query items.
 22. The computer program productof claim 21, further comprising computer instructions that, whenexecuted by the information handling system, cause the system to displayquery items by: performing a second ranking analysis of query itemsafter obtaining the user-specified attribute data by computing anupdated current answer confidence metric for the specified query itemsand an updated marginal answer confidence improvement metric for eachunspecified query item in the plurality of query items; and displayingone or more unspecified query items in ranked order based on the updatedmarginal answer confidence improvement metric for each unspecified queryitem.
 23. The computer program product of claim 17, further comprisingcomputer instructions that, when executed by the information handlingsystem, cause the system to repeat the steps of performing the rankinganalysis and displaying the one or more unspecified query items inranked order in response to additional query attribute data for at leastone of the plurality query items to provide an iterative guided approachfor generating answer items.
 24. The computer program product of claim17, where performing the ranking analysis comprises running a cognitiveanalysis comparison of each marginal answer confidence improvementmetric for each unspecified query item in the plurality of query itemsto rank the query items in ranked order from largest to smallestmarginal answer confidence improvement metric.